layout: false class: title-slide, middle, right, main_slide background-image: url(data:image/png;base64,#images/Italy.png) background-size: cover .font200[The Risk Assessment Shiny App]<br> .font150[A case study with `tidyverse`] <br> .font90[SWIM 2022 <br> Marly Gotti, Apple] ??? Welcome! I'll be presenting the risk assessment applicaiton, a web application to assess the quality of R package based on the shiny. We have seen wonderful rpesentations from Roche, Merck, and others on pacakge validation, so im very excited to talk in this direction. Talks: - R Package Validation at Roche by Coline Zeballos - Statistical Analysis and Pathway to a Risk-based Assessment of R packages at Merck KGaA/EMD Serono by Juliane Manitz --- # About Me <br> .pull-left[ <br> <img src="./images/profile-mask.jpg" class="profile" alt="Marly Gotti"> ] .pull-right[ <br> - Pure mathematician by training - Data scientist at Apple - Former RStudio intern in the `tidymodels` team - Love anything `R` - Executive committee member of the R-Validation-Hub ] ??? I am also an executive committee member of the R Validation Hub. R Validation Hub is a collaboration to support the adoption of R within a biopharmaceutical regulatory setting. --- # Takeaways ??? This presentation is very short, so I have summarized the information you must take away from it -- Open source tools developed by the R-Validation-Hub and the R community ??? The RVH together with the R community have been working on open source tools to help you assess the quality of R packages. -- - `riskmetric` R package ??? assesses the quality of an R pacakge based on a collection of metrics. -- - Risk assessment shiny application ??? web application that extends the functionalities of riskmetric and allows any user, not necesarily R savvy, to use riskmetric and assess the quality of R packages -- Resources - `riskmetric` GitHub: https://github.com/pharmaR/riskmetric ??? This tools are open source. riskmetric can be found in this link. -- - Risk assessment shiny application GitHub: https://github.com/pharmaR/risk_assessment -- - R Validation Hub: https://www.pharmar.org ??? here is the RVH site. The RVH is a collaboration founded by the R-Consortium with the objective to support the adoption of R within a regulatory setting. in the this site, there is a compendium of information on these tools, the philosophy behind them, and articles related to validation you may find useful -- Reach out! ??? If you would like to collaborate, contribute, have any questions, contact me or the team. --- background-image: url("data:image/png;base64,#images/packageTypes.png") background-size: 20% background-position: 80% 40% # Core vs. Contributed R Packages White paper: *"A Risk-based Approach for Assessing R package Accuracy within a Validated Infrastructure"* by Andy Nicholls, Paulo R. Bargo, and John Sims, on behalf of the R Validation Hub. ??? Andy from GSK Paulo from Janssen John from Pfizer The aim of this paper is to propose a possible risk-based approach for assessing R package accuracy within a validated infrastructure. -- - **Core (Base R + Recommended)** - Low risk - See [R: Regulatory Compliance and Validation Issues <br> A Guidance Document for the Use of R in Regulated <br> Clinical Trial Environments](https://www.r-project.org/doc/R-FDA.pdf). ??? R packages that are part of the official R distribution, as formally released by the R Foundation. This software is commonly referred to as “Base R plus Recommended Packages”. The r core team controls R like any of the companies that produces the commercial software that you can buy in this space. Therefore, core packages are considered to be low risk. -- - **Contributed** - Variable risk - Many different authors - Varying SDLCs - Varying levels of popularity - Potentially lots of unknowns ??? 17000 The risk depends - Varying software development life cycles - How well is the package developed / tested? - How much additional exposure has it had within the user community? In the white paper we propose an approach to assess the accuracy of those contributed packages. Once someone in your organization wants to use one of these contributed packages, what steps can you take to start to assess the risk of those packages -- We propose a risk-based approach to establish accuracy/validity for contributed packages. ??? --- # Risk Assessment Framework We propose to assess the risk of contributed R packages based on four criteria: - Purpose: - What will the package be used for? - Statistical vs non-statistical? - How difficult is it for an end user to identify errors? - Maintenance Good Practice (Software Development Life Cycle, SDLC) - How are bugs managed? - What is the documentation like? - Is the source code on GitHub? - Community Usage - Increased exposure to users (community usage) helps to reduce risk - Testing (also part of an SDLC) ??? So once we have decided that we want to use a package the next step that we propose is to assess a package based on 4 criteria - Pu pose: - When talking about purpose, it is assumed that any package that we are talking about is intended to be uses within a GxP workflow, so packages directly manipulating data, creating graphics, or running a statistical model that forms part of that submission. In terms of risks any package should be weighted equally, but we do recognize that modeling packages deserved special attention. There is an innate difficulty to identify errors in those packages vs. those packages that manipulate data. For instance, if you would like to test the arrange function of dplyr, it is easy to do: you can use a small dataset to test this function. On the other hand, statistical models are harder to test. An statistician can roughly check that a model is correct, but not correct to the decimal places. - Development and **maintenance** best practice are essential for any software - How are bugs managed? - What is the documentation like? - Is the source code on GitHub? - Increased exposure to users (**community usage**) helps to reduce risk. Even if a package has no tests but it is is popular, it has been tested by the community. - **Testing** is a crucial part of an SDLC and another specific maintenance best practice metric. --- # Proposed Workflow <br><br> <img src="images/Assessing-package-accuracy.png" alt="source: Assessing Package Accuracy"> --- # The **riskmetric** package - Some questions of interest: - Is the package on CRAN? - Does it have unit tests? - What is its test coverage? - Does the package have documentation? - How many times has the package been downloaded? - Who are the authors? - `riskmetric`: Measures the risk of using an R package. It is calculated based on a number of metrics meant to evaluate development best practices, code documentation, community engagement, and development sustainability. This risk does *not* represent - the risk of damaging the system in which the package is installed, or - the risk that the internal (statistical) functions are correct. - Example: - low risk package: `ggplot2` (risk = 0.206) --- # The Risk Assessment Shiny Application - Framework to provide user feedback on the risk calculated by riskmetric - Inherits the advantages of shiny: no need to code, interactive way to measure risk - Allows authorized personnel to comment on the metrics - Historical comments - Reporting tool - Adopting R within a GxP server ??? We separate three steps in the workflow to assess the risk of an R package using riskmetric: - Finding a source for package information (installed package or CRAN/git source) pkg_ref() - Assessing the package under validation criteria pkg_assess() - Scoring assessment criteria pkg_score() The results will be assembled in a dataset of validation criteria containing an overall risk score for each package as shown in the example below. --- layout: false class: inverse, middle, center # An example using <br> the **Risk Assessment App** [`https://github.com/pharmaR/risk_assessment`](https://github.com/pharmaR/risk_assessment) <video width="320" height="240" controls> <source src="data:image/png;base64,#video/app_final.m4v" type="video/mp4"> Your browser does not support the video tag. </video> --- # Acknowledgements - The R Validation Hub (PSI, EFR Consortium) - Contributors to the `riskmetric` package - Contributors to the Risk Assessment App Thank you for your attention!